Feat/multi git sync by Maleware · Pull Request #729 · stackabletech/airflow-operator

Maleware · 2026-01-04T15:55:38Z

Description

Part of #721

This PR adds support for multiple instances of git-sync.

This is done via a symlink onto the current folder of each git-sync instances.

In kubernetesExecutors a cp -r replaces the symlink as it is a one off anyways and it's easier to transition contents via volumeMounts and init-container.

Integration test 'mount-dags-gitsync' has been changed to use two git-sync resources at different branches of the same repository.

Definition of Done Checklist

Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
Please make sure all these things are done and tick the boxes

Author

Changes are OpenShift compatible
CRD changes approved
CRD documentation for all fields, following the style guide.
Helm chart can be installed and deployed operator works
Integration tests passed (for non trivial changes)
Changes need to be "offline" compatible
Links to generated (nightly) docs added
Release note snippet added

Reviewer

Code contains useful comments
Code contains useful logging statements
(Integration-)Test cases added
Documentation added or updated. Follows the style guide.
Changelog updated
Cargo.toml only contains references to git tags (not specific commits or branches)

Acceptance

Feature Tracker has been updated
Proper release label has been added
Links to generated (nightly) docs added
Release note snippet added
Add type/deprecation label & add to the deprecation schedule
Add type/experimental label & add to the experimental features tracker

…w-operator into feat/multi-git-sync

Maleware · 2026-02-27T12:36:16Z

All tests passed locally apart from

--- FAIL: kuttl/harness/logging_airflow-3.1.6_openshift-false_executor-kubernetes (801.68s)
--- FAIL: kuttl/harness/logging_airflow-3.1.6_openshift-false_executor-celery (1022.60s)

which is a known issue and will be tackled separately.

Maleware · 2026-02-27T12:47:36Z

RELEASE-NOTE-SNIPPET:

Airflow supports now multiple git-sync instances and thus multiple git repositories ( or multiple branches from one repository ). Note that the default path has changed while using git-sync to /stackable/app/allDAGs/current-{i}

…w-operator into feat/multi-git-sync

adwk67

Just a couple of initial comments after running the tests.

adwk67 · 2026-03-03T11:45:59Z

tests/templates/kuttl/mount-dags-gitsync/31-assert.yaml.j2

-  # will expect 2 (two containers, base and gitsync)
-  - script: kubectl -n $NAMESPACE get cm airflow-executor-pod-template -o json | jq -r '.data."airflow_executor_pod_template.yaml"'  | grep "AIRFLOW_TEST_VAR" | wc -l | grep 2
-  # will expect 1 (one container, gitsync)
+  # will expect 6 (2 from from the volume declaration + mounts to four containers, base and gitsync-0, gitsync-1 and multi-gitsync)


I only get 5 here: 3 volume mounts (base, git-sync-0, git-sync-1) and 2 from volume declaration. Likewise, probably, with the others.

adwk67 · 2026-03-03T12:05:12Z

tests/templates/kuttl/mount-dags-gitsync/dag_metrics.py

 # activate DAG
-response = requests.patch(
-    f"{rest_url}/dags/{dag_id}", headers=headers, json={"is_paused": False}
+response_0 = requests.patch(


This step fails for me as the DAGs are imported with errors:

Traceback (most recent call last): File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed File "/stackable/app/allDAGs/current-0/mount-dags-gitsync/dags_airflow3/pyspark_pi.py", line 41, in <module> with open(yaml_path, "r") as file: ^^^^^^^^^^^^^^^^^^^^ FileNotFoundError: [Errno 2] No such file or directory: '/stackable/app/allDAGs/current-0/dags_airflow3/pyspark_pi.yaml'

Although this path does exist.
Tested with this:

[stackable@airflow-webserver-default-0 ~]$ airflow dags list-import-errors /stackable/app/lib64/python3.12/site-packages/airflow/providers/common/compat/security/permissions.py:30 RemovedInAirflow4Warning: The airflow.security.permissions module is deprecated; please see https://airflow.apache.org/docs/apache-airflow/stable/security/deprecated_permissions.html bundle_name | filepath | error ============+==========================================================+========================================================================================================================= dags-folder | current-1/mount-dags-gitsync/dags_airflow3/pyspark_pi.py | Traceback (most recent call last): | | File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed | | File "/stackable/app/allDAGs/current-1/mount-dags-gitsync/dags_airflow3/pyspark_pi.py", line 41, in <module> | | with open(yaml_path, "r") as file: | | ^^^^^^^^^^^^^^^^^^^^ | | FileNotFoundError: [Errno 2] No such file or directory: '/stackable/app/allDAGs/current-1/dags_airflow3/pyspark_pi.yaml' | | dags-folder | current-0/mount-dags-gitsync/dags_airflow3/pyspark_pi.py | Traceback (most recent call last): | | File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed | | File "/stackable/app/allDAGs/current-0/mount-dags-gitsync/dags_airflow3/pyspark_pi.py", line 41, in <module> | | with open(yaml_path, "r") as file: | | ^^^^^^^^^^^^^^^^^^^^ | | FileNotFoundError: [Errno 2] No such file or directory: '/stackable/app/allDAGs/current-0/dags_airflow3/pyspark_pi.yaml' | | [stackable@airflow-webserver-default-0 ~]$

I notice you have a different folder structure in your repo. In the one used for the tests we have:
gitFolder: "mount-dags-gitsync/dags_airflow3" and the dags in the pods are here:
/stackable/app/allDAGs/current-0/mount-dags-gitsync/dags_airflow3/ which doesn't match /stackable/app/allDAGs/current-0/dags_airflow3/pyspark_pi.yaml

And I think this is brittle:

yaml_path = os.path.join( os.environ.get("AIRFLOW__CORE__DAGS_FOLDER"), "current-1", "dags_airflow3" ,"pyspark_pi.yaml" )

as the DAG itself has to know about the nestedness.

Ah yeah, it slipped through. We'd need to teach the whole path to the DAG in the repository I guess.

I understand. We could introduce new env-variables which could be set via the operator like GIT_0 and GIT_1 pointing to the respective repositories. Then instead we could just

yaml_path = os.path.join( os.environ.get("GIT_1") ,"pyspark_pi.yaml" )

The user would set the path-to-folder ( "mount-dags-gitsync/dags_airflow3" in this case ) and we prefix it with the AIRFLOW_DAG_FOLDER path.

For integration tests I think it would maybe be fine to reference like it is currently?

I'm trying with this which seems to work:

#yaml_path = os.path.join( # os.environ.get("AIRFLOW__CORE__DAGS_FOLDER"), "current-0", "dags_airflow3" ,"pyspark_pi.yaml" #) dags_folder = Path(os.environ.get("AIRFLOW__CORE__DAGS_FOLDER", "/stackable/app/allDAGs")) / "current-0" matches = list(dags_folder.rglob("pyspark_pi.yaml")) if not matches: raise FileNotFoundError(f"pyspark_pi.yaml not found under {dags_folder}") if len(matches) > 1: raise RuntimeError(f"Multiple pyspark_pi.yaml files found under {dags_folder}: {matches}") yaml_path = matches[0]

adwk67 · 2026-03-03T13:00:43Z

docs/modules/airflow/pages/usage-guide/mounting-dags.adoc

+
+If you want to access multiple branches of a repository or load dags from multiple repositories, you can extend the list shown above.
+
+The default mount-path for git-sync resources is `/stackable/app/allDAGs/current-{i}` where i corresponds to:


I don't think the user needs to know this as it is (or should be?) an internal detail.

adwk67 · 2026-03-03T13:01:57Z

docs/modules/airflow/pages/usage-guide/mounting-dags.adoc

+          sshPrivateKeySecretName: git-sync-ssh
+----
+
+NOTE: Using DAG's from ConfigMaps will need you to either use celeryExecutors and mount them under `/stackable/app/allDAGs/<name>` or use kubernetesExecutor and mount them somewhere else and change the PYTHONPATH as well as the `AIRFLOW\__CORE__DAGS_FOLDER` accordingly.


Suggested change

NOTE: Using DAG's from ConfigMaps will need you to either use celeryExecutors and mount them under `/stackable/app/allDAGs/<name>` or use kubernetesExecutor and mount them somewhere else and change the PYTHONPATH as well as the `AIRFLOW\__CORE__DAGS_FOLDER` accordingly.

NOTE: Using DAGs from ConfigMaps will require you to either use celeryExecutors and mount them under `/stackable/app/allDAGs/<name>` or use the kubernetesExecutor and mount them somewhere else, changing the PYTHONPATH (if using submodules) and `AIRFLOW\__CORE__DAGS_FOLDER` accordingly.

adwk67

Is this a breaking change? Will existing set-ups (DAGs mounted via CMs, single gitsync repos) still work? The versioning test (also using gitsync) does not pass for me (probably as it is still referencing the dag in the mani branch).

adwk67 · 2026-03-03T13:41:01Z

rust/operator-binary/src/crd/mod.rs

+        symlinks
+    }
+
+    // kubernetesExecuter needs copy since it needs init-container.


Can we give a reason for this (I think it was something to do with the symlink not resulting in "clean" paths, or something...)

adwk67 · 2026-03-03T13:41:22Z

rust/operator-binary/src/crd/mod.rs

                    container_debug_command(),
                    "airflow triggerer &".to_string(),
                ]),
+                // This essentially doesn't work for KubernetesExecutor


Can we give a reason?

adwk67 · 2026-03-03T13:42:28Z

rust/operator-binary/src/crd/mod.rs

            format!(
                "cp -RL {CONFIG_PATH}/{AIRFLOW_CONFIG_FILENAME} {AIRFLOW_HOME}/{AIRFLOW_CONFIG_FILENAME}"
            ),
+            // Adding cm as dags within the same AIRFLOW_DAGS_FOLDER may lead to problems, thus checking if exists.


Does mean that we can't simply have, say, 2 DAGs in a single ConfigMap, which are mounted to AIRFLOW_DAGS_FOLDER, or something else?

sbernauer and others added 28 commits December 23, 2025 10:50

feat: Support objectOverrides

f939934

update to newer op-rs

d6b4c95

changelog

49848cf

Adding support for multiple git-sync repos

fe99463

Merge branch 'main' into feat/multi-git-sync

65e39a9

Cleaning up

2cbae70

Merge branch 'feat/multi-git-sync' of github.com:stackabletech/airflo…

3ea363a

…w-operator into feat/multi-git-sync

makeing pre-commit happy

26c11a7

Adding check if cm is mounted into core folder, if omit mkdir

7c7f25f

store wip

ffa249a

working example, clean up

984d7c1

Merge branch 'main' into feat/multi-git-sync

5021b33

update to v1alpha2

6538c67

Add CHANGELOG.md

0e0b09f

Adding docs draft

04ec315

Better documentation

3e2c438

Refactor strings into constants

010bfd3

Further removing strings and consolidate in constant

50427cb

Improve comments and refactor

a54f9e5

Add two more comments

8b2f4b6

Another comment

61dc8ce

removing add_env for dags-init container

7e80e9a

consistent names across init containers

d9558ea

add only if git-sync is not empty in pythonpath

c5befaf

Updating integration test mount-dags-gitsync to use two git-syncs

c2b6e44

making pre-commit happy again

cf8b245

again, pre-commit for dag-metrics.py

1c188c5

rework typos

db8ecbd

better documentation for multi-git-sync

8db36d6

Merge branch 'main' into feat/multi-git-sync

c7052a8

Maleware self-assigned this Feb 27, 2026

Maleware added 2 commits February 27, 2026 14:04

again pre-commit hook was unhappy

1b43f0d

Merge branch 'feat/multi-git-sync' of github.com:stackabletech/airflo…

bd8d7a4

…w-operator into feat/multi-git-sync

Maleware mentioned this pull request Feb 27, 2026

Git-sync: Support multiple repositories in one AirflowCluster #721

Open

Maleware requested a review from adwk67 February 27, 2026 13:19

Add test print I forgot

ff8bb20

adwk67 added this to Stackable Engineering Mar 3, 2026

adwk67 moved this to Development: In Review in Stackable Engineering Mar 3, 2026

adwk67 reviewed Mar 3, 2026

View reviewed changes


		If you want to access multiple branches of a repository or load dags from multiple repositories, you can extend the list shown above.

		The default mount-path for git-sync resources is `/stackable/app/allDAGs/current-{i}` where i corresponds to:

	NOTE: Using DAG's from ConfigMaps will need you to either use celeryExecutors and mount them under `/stackable/app/allDAGs/<name>` or use kubernetesExecutor and mount them somewhere else and change the PYTHONPATH as well as the `AIRFLOW\__CORE__DAGS_FOLDER` accordingly.
	NOTE: Using DAGs from ConfigMaps will require you to either use celeryExecutors and mount them under `/stackable/app/allDAGs/<name>` or use the kubernetesExecutor and mount them somewhere else, changing the PYTHONPATH (if using submodules) and `AIRFLOW\__CORE__DAGS_FOLDER` accordingly.

Uh oh!

Conversation

Maleware commented Jan 4, 2026 • edited by sbernauer Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Definition of Done Checklist

Author

Reviewer

Acceptance

Uh oh!

Maleware commented Feb 27, 2026

Uh oh!

Maleware commented Feb 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adwk67 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adwk67 Mar 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Maleware Mar 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adwk67 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Maleware commented Jan 4, 2026 •

edited by sbernauer

Loading

Maleware commented Feb 27, 2026 •

edited

Loading

adwk67 Mar 3, 2026 •

edited

Loading

Maleware Mar 3, 2026 •

edited

Loading